Commentary: Measuring what matters: the case for patient generated PROMS.

When I started researching amyotrophic lateral sclerosis (ALS) in 2002 I was trained to interview patients and administer a 12 item scale assessing their ability to walk, speak, and breathe. It yielded a score between 48 (healthy) and 0 (incapacitated). This functional rating scale (ALSFRS-R) was developed to support clinical trials and correlates with survival and markers of neurodegeneration. 3 In contrast to the shared notes schemes Nelson and colleagues’ describe, we were not allowed to tell patients what their score was. The scale was seen as a research tool to describe groups, not the progress of individuals, so would it help or harm patients to know they scored 23 or 35? Subsequent validation studies showed the measure was just as reliable if administered by caregivers, by telephone, or by patients themselves. In November 2005 patients were given access to an online self reported version of the scale and visualisations of their progress compared with that of other patients on PatientsLikeMe.com. This website was built by a family affected by ALS who wanted to learn more about the progression of the disease and manage it effectively. Self reported data submitted to the website were validated against clinical measures and used to help understand the effect of treatments. Crowdsourced models of ALSFRS-R scores recorded by clinicians have been used to predict outcomes, and patient reported scores might achieve the same goal. Tellingly, the predictive algorithms designed by mathematically minded solvers who were non-experts in the disease substantially outperformed the predictions of ALS expert clinicians. Regrettably, few other groups of patients have such well developed patient reported outcome measures (PROMs) for their particular conditions. Done well, such tools can serve so many purposes: identifying treatment effects, mapping to pathology, supporting predictive modelling for researchers, guiding clinical care, and offering feedback to support self management. In the absence of a common framework to systematically evaluate PROMs against these purposes many who develop them continue to produce tools that fulfil only a subset of these ambitions. Part of the problem is that clinicians often lack psychometric expertise or the time and funding to develop more useful tools. As a result, many PROMs are sponsored by the pharmaceutical industry, in line with Food and Drug Administration guidelines, to support label claims for the added value of their medicines to improve symptom relief or quality of life. Some claim that an unintended consequence of this has been the development of a “cottage industry” of companies developing measures with the needs of pharmaceutical sponsors primarily in mind. Furthermore the sharing of such tools may be restricted through licensing because the measurements are considered to offer a competitive advantage. Alternative paths to systematically developing PROMs are, however, being explored. The PROMIS initiative, a group of academic researchers, develops measures that use computerised adaptive testing (CAT). Unlike traditional measures, which use a fixed list of items, CAT adjusts the level of questions according to the patient’s previous answers. CAT shows good performance against traditional measures, but understanding responses requires digital administration and new skills, and, crucially, no PROMIS instrument has yet been used as an endpoint to support an FDA label claim. From the patient side, “patient powered research networks” such as PCORnet (www.pcornet.org/) offer the opportunity for large online registries of patients to contribute to the psychometric development of new PROMs. On another network, PatientsLikeMe, the Open Research Exchange (www. openresearchexchange.com) lets researchers rapidly create and validate measures with substantial patient input by combining an engaged patient network with online software that guides researchers through the process of developing PROMs with constant patient feedback. Instruments developed on the Open Research Exchange are licensed under Creative Commons, meaning they are free for use and adaptation, and work is under way to understand how PROMs developed online might meet FDA guidelines in a reliable and resource effective manner. Measures that are more patient centred might frame questions in a more positive and aspirational manner, use the patient’s own language, and harness CAT to be brief and responsive,